Scott Aaronson's Shtetl-Optimized journal is almost always thoughtful and entertaining. Yesterday he comments most engagingly on the challenge people have of evaluating low-probability but high-impact events — in this case, whether high-energy physics experiments could cause the catastrophic destruction of the Earth, or maybe the entire Universe [1]. Aaronson mentions "salience bias", the fascinating tendency humans have to worry about dramatic things (explosions, disasters, big fierce animals, etc.), rather than to objectively evaluate the odds. In a follow-up comment Scott observes:
I think it's easy to fall victim to "premature Bayesianism": that is, trying to be rigorous by demanding probabilities for specific astronomically unlikely events, while implicitly assigning many other related events a probability of 0 because you haven't even considered them. Economists might consider this an instance of salience bias. From my perspective, though, something like it is probably inevitable when computationally-bounded agents like ourselves try to simulate Bayesian rationality. We're never going to succeed, since the space of potentially-relevant events is exponentially large, and summing over them would be #P-complete even if we knew the right prior.
and in a later remark:
As a side note, it's always struck me how people get more worked up about civilization being destroyed by grey goo or malevolent AI-bots or particle physics disasters, than they do about its destruction by completely non-hypothetical methods: say chopping down all the forests, filling the oceans with garbage and the atmosphere with billions of years' worth of accumulated carbon. Maybe the fact that the real dangers are (relatively) slow creates a false sense of security, or maybe the fact that they're real makes them less fun to worry about.
(I've written about this sort of thing here sporadically — cf. NoiseAndPredictability (1999-09-14), BiggerPictures (1999-11-22), LoomingDisaster (2001-08-06), ProbabilisticTragedy (2003-03-12), IllusionOfControl (2004-10-21), ... — and for an earlier hat-tip to "Shtetl-Optimized" see Beware the Breakthrough (2008-01-07)) - ^z - 2008-06-22
(correlates: Beware the Breakthrough, BirdWhile, WeHappyFew, ...)